Method of image processing for an augmented reality application

ABSTRACT

An apparatus for and method of image processing in an augmented reality application is provided. The method includes the steps of: providing at least one image of a real environment; performing image processing in an augmented reality application with the at least one image employing visualization of overlaying digital information with visual impressions or the image of the real environment and employing vision-based processing or tracking; and adjusting at least one of a parameter and operating flow of the vision-based processing or tracking depending on at least one of the following: a usage of the image processing, a usage of the visualization, a visually perceivable property of the digital information or the real environment, a property of a display device employed in the visualization, or a manner in which a user is viewing the visualization.

This application is entitled to the benefit of, and incorporates byreference essential subject matter disclosed in PCT Application No.PCT/EP2012/069247 filed on Sep. 28, 2012.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates to a method of image processing in anaugmented reality application, comprising the steps of providing atleast one image of a real environment and performing image processing inan augmented reality application with the at least one image employingvisualization of overlaying digital information with visual impressionsor the image of the real environment and employing vision-basedprocessing or tracking. The invention also relates to a computer programproduct comprising software code sections for performing the method.

2. Background Information

Augmented reality systems and applications present enhanced informationof real environment by providing a visualization of overlaying digitalinformation, particularly computer-generated virtual information, withvisual impressions or an image of the real environment. The digitalinformation can be any type of visually perceivable data such asobjects, texts, drawings, videos, or any combination thereof. The realenvironment is captured, for example, with a camera held by a user orattached on a device held by a user. The digital information has to besuperimposed with the real environment or a part of the real environmentin the camera image at a right time, at a right place and in a right wayin order to offer a satisfied visual perception to users.

The right time requires that the digital information should besuperimposed with the real environment in the image when and only whenit is required or necessary, for example, when a particular real objectappears in the camera image (e.g., See Kato, H. and M. Billinghurst.Marker Tracking and HMD Calibration for a Video-based Augmented RealityConferencing System. in 2nd IEEE and ACM International Workshop onAugmented Reality (IWAR 99). 1999. San Francisco, Calif.; referred tohereinafter as “Kato”), or when the camera is positioned at a particulargeometrical location (e.g., See S. Feiner et al., “A Touring Machine:Prototyping 3D Mobile Augmented Reality Systems for Exploring the UrbanEnvironment,” Proc. 1st Int'l Symp. Wearable Computers (ISWC'97), IEEECS Press, Los Alamitos, Calif., 1997, pp. 74-81).

The digital information should be superimposed with the real environmentat desired pixel positions within the image, for example in a correctperspective way, i.e. adapted and derived from the real environmentbeing viewed. In order to achieve this, the pose of the camera, i.e.orientation and position, with respect to the real environment or a partof it has to be known (e.g., See Kato). Vision is an indispensablecomponent for computing the camera pose in augmented realityapplications, as the camera image that captures the real environment canalways be a means for camera pose estimation and\or real objectdetection. Various vision-based online tracking solutions have beendeveloped to compute the pose of the camera for augmented realityapplications; e.g., See Sanni Siltanen, Theory and applications ofmarker-based augmented reality. Espoo 2012. VTT Science 3; which mayalso be found at http://www.vtt.fi/inf/pdf/science/2012/S3.pdf;hereinafter referred to as “Siltanen”).

The right way means that the digital information should be embedded intothe real environment in the image or view of the real environment (forexample, viewed by an optical see-through display device) depending onthe purpose of an application. For augmented reality applications wherevirtual information is used to give instructions, draw users' attention,or support the understanding of 3D shapes and dimensions, the virtualinformation is preferred to be superimposed with real environment suchthat it is bright and indistinguishable from the real environment.Examples of such applications are augmented assembly and maintenancesupport; e.g., See Azpiazu, J., Siltanen, S., Multanen, P., Makiranta,A., Barrena, N., Diez, A., Agirre, J. & Smith, T. Remote support formaintenance tasks by the use of Augmented Reality: the ManuVAR. Thisvisualization is so called non-photorealistic visualization. Incontrast, photorealistic visualization is preferred in many applicationswhere virtual information is visually indistinguishable from the realenvironment. Virtual interior design (e.g., See Sanni Siltanen andCharles Woodward. 2006. Augmented interiors with digital camera images.In Proceedings of the 7th Australasian User interface conference—Volume50 (AUIC '06), Wayne Piekarski (Ed.), Vol. 50. Australian ComputerSociety, Inc., Darlinghurst, Australia, Australia, 33-36; hereinafterreferred to as “Siltanen and Woodward”) and augmented clothing areexamples of such applications. For this, many solutions have beendeveloped to have an enhanced visual realism by incorporating occlusion(e.g., See in Ivan J. Jaszlics, Sheila L. Jaszlics, U.S. Pat. No.6,166,744, System for combining virtual images with real-world scenes;hereinafter referred to as “Jaszlics”), illumination (e.g., See Siltanenand Woodward), noise and motion blur in camera images (e.g., See JanFischer, Dirk Bartz, and Wolfgang Strasser. Enhanced visual realism byincorporating camera image effects. In Proceedings of the 5th IEEE andACM International Symposium on Mixed and Augmented Reality (ISMAR '06).2006; hereinafter referred to as “Fischer”) into the visualization. Amost recent survey of augmented reality technologies, includingtracking, visualization, user interface, etc.; e.g., as presented inSiltanen.

Many visualization techniques for augmented reality applications havebeen developed to improve visualization of overlaying digitalinformation with visual impressions or an image of the real environment,particularly the visual perception of the overlay of digital informationand real environment. In order to have a photorealistic visualization,Siltanen and Woodward erases artificial markers that are used for camerapose estimation from images and introduces lighting and shadow effectsto virtual objects. Fischer incorporates camera image effects, e.g.noise, motion blur, to virtual objects. Jaszlics detects the range dataof real environment to generate a correct visual effect of the virtualinformation occluded by the part of real environment in the overlayimage of the virtual information and the real environment.

The performance of vision-based tracking solutions is often quantifiedin terms of re-projection error or the result of similarity measure. There-projection error corresponds to the pixel distance between aprojected point of a real 3D point and a measured one in an image. Thesimilarity measure computes the degree of difference between atransformed reference visual feature and a visual feature in a cameraimage. Common examples of image similarity measures include thesum-of-squared differences (SSD), cross-correlation, and mutualinformation. The result of a similarity is a real number. If thesimilarity measure is the sum-of-squared difference between two imagepatches, the smaller the similarity measure result is, the more similarthe two visual features are. If the similarity measure is the zeronormalized cross-correlation between two image patches, the bigger thesimilarity measure result is, the more similar the two visual featuresare. The proposed method according to the invention, as set out below,can use any of these techniques.

The parameters (including their values) and operating flow or workflowof the vision-based processing or tracking solutions are alwaysconfigured such that the similarity measure and\or the re-projectionerror are minimized; e.g., See Wang, L.; Springer, M.; Heibel, T. H. &Navab, N. (2010), Floyd-Warshall all-pair shortest path for accuratemulti-marker calibration., in ‘ISMAR’, IEEE, pp. 277-278. However, inaugmented reality applications, the challenge is that none of aminimized similarity measure and a minimized re-projection error couldindicate or guarantee a user satisfying overlay of digital informationand real environment in a camera image or view of an optical see-throughdisplay device. Therefore, even when vision-based processing or trackingsolutions are optimized in terms of the similarity measure and\or there-projection error, the problem of user unsatisfied visualization oftenhappens, such as jittering, virtual information misaligned with realenvironment, discontinuous display of virtual information.

Augmented reality authoring tools allow people, who have, e.g., nosoftware development, image processing and computer vision background,to create powerful and flexible augmented reality applications in asimple and intuitive way. The authoring tools decouple the usage of thevisualization and augmented reality application from the vision-basedtracking, detection, or localization solutions. The users of theauthoring tools have the knowledge of the usage of the visualization andthe properties of the digital information to be overlaid or of the realenvironment. Only the reference object used for localizing the imagingsensor (camera) in the environment is considered when selecting the typeand the parameters of the vision-based tracking algorithms. However, asdiscussed above, the augmented reality visualization could be disturbedeven when the tracking solutions are optimized for the similaritymeasure and\or the re-projection error.

It would therefore be beneficial to provide a method of image processingin an augmented reality application which is capable to improve theperformance and usability of augmented reality applications,particularly augmented reality authoring tools.

SUMMARY OF THE INVENTION

According to an aspect of the invention, there is provided a method ofimage processing in an augmented reality application, comprising thesteps of providing at least one image of a real environment, performingimage processing in an augmented reality application with the at leastone image employing visualization of overlaying digital information withvisual impressions or the image of the real environment and employingvision-based processing or tracking, and adjusting at least one of aparameter and operating flow of the vision-based processing or trackingdepending on at least one of the following: a usage of the imageprocessing, a usage of the visualization, a visually perceivableproperty of the digital information or the real environment, a propertyof a display device employed in the visualization, a manner in which auser is viewing the visualization.

Accordingly, the present invention is capable to improve the performanceand usability of augmented reality applications, such as authoringtools. Particularly, the invention attends to the problem of userunsatisfied visualization of overlaying the virtual information ordigital content to an image or view of a real environment in augmentedreality or visual search applications that employ vision-baseddetection, localization or tracking solutions by adjusting parametersincluding their values and/or operating flow (workflow) of thevision-based processing or tracking according to the usage of the imageprocessing or the visualization, according to a visually perceivableproperty of the digital information and/or the real environment,according to display devices, and/or according to the way of userviewing the visualization. For example, it is proposed to adapt thechoice of the type and/or the parameters of the vision-based processingor tracking methods according to one or a set of additional input likethe usage of the processing or tracking methods, the purpose of thevisualization, model specifications of the digital information and/orthe real environment, etc. None of the above described visualizationtechniques in the prior art takes the performance or the process oftracking solutions into consideration.

According to an embodiment, the vision-based processing or trackinginvolves a method for vision-based tracking, detection, localization, orclassification.

For example, the parameter of the vision-based processing or trackingincludes a variable having a value, a number of features in the imagematched with a set of reference features, or a number of features in theimage to be used for a pose estimation of a camera capturing the image.

The features are, but not limited to, intensities, gradients, edges,lines, segments, corners, descriptive features or any other kind offeatures, primitives, histograms, polarities or orientations.

According to an embodiment, the operating flow of the vision-basedprocessing or tracking includes components and a combination of thecomponents in the process of vision-based processing or tracking,wherein the components are sensors measuring a spatial property of anobject or are mathematical methods used in computer vision or imageprocessing field. For example, the spatial property is any propertyrelating to or occupying a space, such as dimensionality,directionality, location, geometry. Preferably, the geometry describesat least one of the following attributes: shape, symmetry, planarity,geometrical size, density. The mathematical methods may include at leastone of the following: a linear equation solver, a cost function, anoptimization method.

According to an embodiment, the usage of the image processing includesat least one of the following: detecting or recognizing the realenvironment or a part of it, visual search, classification problems,estimating a pose of a capturing device capturing the image, trackingthe real environment or a part of it.

According to a further embodiment, the usage of the visualizationincludes at least one of the following: delivering user's informationassociated to the real environment or a part of it, enhancing user'svisual perception of the digital information and/or the real environmentby superimposing the digital information to the real environment in acorrect perspective.

According to an embodiment, the visually perceivable property of thedigital information or the real environment includes at least one of thefollowing: a spatial property, symmetry, behavior, color, type of thedigital information or the real environment. For example, the behaviorincludes at least one of the following: static, moving, a way of moving,and/or the type includes at least one of the following: a text, adrawing, a video.

Particularly, the display device is a device displaying the overlay ofthe digital information and the real environment to a user, such as amonitor or an optical-see-through head mounted display.

According to an embodiment, the manner in which a user is viewing thevisualization includes eye gazing or eye focusing. For example, only thepart of the digital information which is in the user's focus may beoverlaid precisely, whereas any other part of the digital informationmay be processed differently. For instance, eye gazing or eye focusingmay be detected by an eye tracking sensor.

The invention may further include the step of employing in thevision-based processing or tracking a similarity measure ofsum-of-squared differences, wherein the similarity measure is at leastin part defined by a threshold value, and adjusting the threshold value.

For example, in the vision-based processing or tracking, at least oneparameter describing a translation of the digital information isoptimized, and at least one parameter describing a rotation around thedigital information is not considered.

According to an embodiment, in a visualization of a digital informationundertaking a certain animation that consists in a complex movement ortrajectory of the digital information, a stable pose estimation of acamera capturing the image is not performed in the vision-basedprocessing or tracking. For example, a number of feature correspondencesemployed to estimate the camera pose is decreased.

According to an embodiment, a threshold of a re-projection error of thedigital information is adjusted according to a pixels-per-inch (PPI)parameter of the display device.

The method according to the invention is performed on a computer system,such as a stationary or mobile computer system, preferably on handhelddevices as mentioned above.

The invention also relates to a computer program product adapted to beloaded into the internal memory of a digital computer system, andcomprising software code sections by means of which the steps accordingto any of the above aspects and embodiments are performed when saidproduct is running on said computer system.

BRIEF DESCRIPTION OF THE DRAWINGS

Further aspects, advantageous features and embodiments of the inventionwill be evident from the following description in connection with thedrawings.

FIG. 1 shows a flowchart of an exemplary augmented reality visualizationapproach according to the state of art.

FIG. 2 shows a flowchart of a method according to an embodiment of theinvention.

FIG. 3 shows a flowchart of a method according to an embodiment of theinvention overlaying a virtual spherical object onto the image of thereal environment based on an embodiment of the present invention.

FIG. 4 shows a flowchart of a method according to an embodiment of theinvention overlaying a virtual object (here sofa) onto the image of thereal environment in augmented interior design application based on anembodiment of the present invention.

FIG. 5 shows a flowchart of a method according to an embodiment of theinvention of playing a video in full screen mode in an augmented movieposter application based on an embodiment of the present invention.

FIG. 6 shows a flowchart of a method according to an embodiment of theinvention overlaying a virtual tree leaf onto the image of the realenvironment based on an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows a flowchart of an exemplary augmented reality visualizationapproach according to the state of art. In step 1 an image of realenvironment is captured by a capturing device, particularly a camera. Instep 2.a, vision-based processing and/or tracking is performed on thecaptured image. The virtual information provided in step 2.b is overlaidonto the image of the real environment in step 3. In anotherapplication, the virtual information provided in step 2.b may beoverlaid onto the view of the real environment which is provided by anoptical-see-through display device, as commonly known in the art.

The visualization of overlaying digital (e.g., virtual) information ontothe image of a real environment could be used for different purpose.Augmented reality visualization is often used to enhance user's visualperception by superimposing virtual information or digital content tothe real environment in a correct perspective. For example, augmentedinterior design could superimpose a virtual sofa to the camera image ofthe real environment of the room in a correct perspective: e.g., asdisclosed in Siltanen and Woodward. This application could visuallysupport furniture arrangement in the room without having to move themphysically. For this, an accurate pose of the camera with respect to theroom is mandatory.

FIG. 2 shows a flowchart of a method of image processing according to anembodiment of the invention. The result of the step 2.b of preparingvirtual information as the digital information to be overlaid will be aninput to the step of performing vision-based processing or tracking instep 2.a through steps 2.c and 2.d. The step 2.b of providing thedigital information describes the process of providing the digitalinformation, its visual representation, its usage, and/or its visuallyperceivable property.

In addition to the steps described with reference to FIG. 1, in thepresent embodiment the property of the virtual information, such assymmetry, behavior, color, etc., and/or its usage may be checked orinput by the user in step 2.c, wherein the method in response to thisinformation configures or adjusts the processing or tracking of theimage processing method in step 2.a accordingly. For example, in step2.d parameters of the vision-based processing and/or tracking accordingto the result of 2.c are adjusted. In step 3, the virtual information isthen overlaid onto the image of the real environment. In step 2.e, thetype of augmented reality (AR) application, a scenario definition, acontext, etc. may be provided to the method as additional input.

FIG. 4 shows a flowchart of a method according to an embodiment of theinvention overlaying a virtual object (here a furniture, like a sofa)onto the image of the real environment in an augmented interior designapplication based on an embodiment of the present invention demonstratedin FIG. 2. The property of the virtual information, such as symmetry,behavior, color, etc., and/or its usage is checked in step 2.c, whereinthe method in response to this information configures or adjusts theprocessing or tracking method accordingly. For example, in step 2.d thethreshold value in the similarity measures of sum-of-squared differencesis increased. In step 3, the virtual sofa is then overlaid onto theimage of the real environment.

Augmented reality visualization can also be used to convey user'sinformation associated to the real environment or a part of it, as longas the real environment or the part of it is seen by a pointing device,such as a camera. For example, when a piece of art in the realenvironment is seen by a camera image, texts regarding the art's historymay be superimposed onto the image. In the example of an augmented filmposter, a trailer will be played as soon or as long as the poster isseen by the camera image. In these situations, the virtual informationor the digital content, i.e. the texts or trailer, often has to besuperimposed on the image in a way that a human can easily read it.Therefore, in this application the exact pose of the camera with respectto the part of real environment, i.e. the art and poster, is often notnecessary. In some applications, where the virtual information ordigital content is displayed on the screen non-registered to the realenvironment, the pose of the camera is even not required at all. Theexample of such application is an augmented film poster where it plays amovie in a full screen mode or triggers displaying a web page on thescreen when the poster is detected in the camera image. However, thehigh recognition rate of detecting the real environment or a part of itin the camera image becomes critical. A low recognition rate couldresult in the problems, such as false positive display of the virtualinformation, or positive false display of the virtual information.

FIG. 5 shows a flowchart according to an embodiment playing a video (asthe digital information to be overlaid, step 2.b) in full screen mode inan augmented movie poster application based on the embodiment of thepresent invention demonstrated in FIG. 2. As a result of step 2.c, asdescribed above, the threshold value in the similarity measures ofsum-of-squared differences could be decreased. In step 3, the video isplayed in full screen mode.

Many vision-based tracking solutions cannot meet both of accurate poseestimation and high recognition rate. For example, a low threshold valuein the similarity measures of sum-of-squared differences could result inan accurate estimated pose, but may lead to a low recognition rate. Onthe other side, a high threshold value for the similarity measures couldresult in an inaccurate estimated pose, but will lead to a highrecognition rate. Therefore, adjusting the value of the thresholddepending on the usage of the augmented reality visualization could leadto a user satisfying visualization.

The visual properties of the virtual information or the real environmentare, but not limited to, the spatial property, behavior, and color.

When virtual information is a spherical object with homogenous textureon its surface, rotating the spherical object around the sphericalcenter will not result in different visual perception to users.Translating the spherical object will result in different visualperception to users. Therefore, in the optimization process, parametersdescribing the translation of the virtual object have to be optimized,while parameters describing the rotations around the spherical do notneed to be considered.

FIG. 3 shows a flowchart according to an embodiment overlaying a virtualspherical object as the digital information onto the image of the realenvironment based on the embodiment of the present inventiondemonstrated in FIG. 2. In step 2.b, a virtual spherical object withhomogenous texture on its surface is provided. As a result of step 2.c,as described before, parameters describing the rotations around thespherical center are adjusted in a way that they are not considered. Instep 3, the virtual spherical object is overlaid onto the image of thereal environment.

In order to have user satisfied visualization of augmented reality,parameters in the vision-based tracking solution may need to be adjustedaccording to the behavior of virtual information or the realenvironment. In the application of augmented interior design, thevirtual information (e.g., sofa) is preferred to be stably superimposedwithout uttering onto the real room in the camera image. For this, anumber of feature correspondences are often needed to have stable camerapose estimation. However, detecting or matching a large number offeature correspondences is challenging and often requires a manualinitialization and processing time. It is suitable for offlineapplications, like superimposing virtual furniture onto a pre-captureimage of the room, but not for online real-time applications.

In other applications, augmenting the environment with virtual objectsundertaking a certain animation that consist in a complex movement ortrajectory, such as a butterfly or a tree leaf falling down, do notrequire a stable camera pose estimation since the jitter of the camerapose estimation can be perceived by the user as being part of theanimation of the virtual object. Another example is superimposingvirtual information onto the part of a real environment in a cameraimage, in which the part of real environment is moved with respect tothe camera. This often requires online real-time camera pose estimationwithout obvious lag, but the jitter of the camera pose estimation is notcritical as it will be visually compensated by the movement of the partof the real environment. In these situations, fewer featurecorrespondences are preferred to estimate the camera pose for areal-time performance.

FIG. 6 shows a flowchart according to an embodiment overlaying a virtualtree leaf (provided in step 2.b as the digital information) onto theimage of the real environment based on the embodiment of the presentinvention demonstrated in FIG. 2. As a result of step 2.c, as describedabove, in step 2.d the number of feature correspondences used toestimate the camera pose could be decreased. In step 3, the virtual treeleaf is overlaid onto the image of the real environment.

According to another embodiment, parameters in the vision-basedprocessing or tracking may be adjusted according to the display devicesused in the visualization for an optimal and user satisfying augmentedreality visualization. One pixel in a monitor having 1 pixels-per-inch(PPI) is one inch, while one pixel in a monitor having 1000 PPI is1/1000 inch. One-pixel misalignment between the digital information tobe overlaid and the real environment or a part of it in the image ismore visually obvious to the human eye when it is displayed on a monitorwith low PPI than a monitor with high PPI. Therefore, the threshold ofthe projection error may be adjusted according to the PPI parameter ofthe display device to have user satisfying visual perception of theaugmented reality visualization.

The present invention thus discloses various embodiments of a method ofimage processing to remove potential visualization problems ofoverlaying digital information and real environment in augmented realityapplications by adjusting parameters (including their values) and/or theoperating flow (or workflow) of vision-based processing or trackingsolutions according to the usage of the image processing or thevisualization, according to visually perceivable property of the virtualinformation or the real environment, according to display devices,and/or according to the way of user viewing the visualization.

Particularly, the potential visualization problems are undesiredvisually perceivable effects, such as uttering of the digitalinformation, misalignment between the digital information and the realenvironment or a part of it, discontinuous display of the digitalinformation, false positive display of the digital information, orpositive false display of the digital information.

Potential vision-based tracking solutions are, but are not limited to,methods or systems for tracking, detection or classification based onvisual information, like 2D or 3D images that could be captured orgenerated by cameras or spatial scanners.

The parameter of a vision-based tracking solution may be a variablehaving a value, such as a threshold for an acceptable result ofsimilarity measure or re-projection error, the number of featuresmatched with a set of reference features or the number of featuresneeded for camera pose estimation.

The value of the parameter could be any data type, such as integer,boolean, binary, real number, char, string, symbol.

The operating flow or workflow of the vision-based processing ortracking may include components and a combination of components in theprocess of vision-based tracking. The components are, but not limitedto, sensors measuring a spatial property of an object or mathematicalmethods used in computer vision and image processing field. The spatialproperty may be any property relating to or occupying a space, such asdimensionality, directionality, location, geometry. Geometry maydescribe the following attributes, but not limited to, shape, symmetry,planarity, geometrical size, and density. The mathematical methods mayinclude, but not limited to, linear equation solver, cost function, andoptimization method. A cost function is a formula that reaches a maximumor minimum as visual features are brought into alignment in the field ofcomputer vision and image processing. A common example of cost functionis a similarity measure formula. An optimization method finds optimalvalues for the arguments of a cost function such that the functionreaches a maximum or minimum by systematically choosing values ofarguments from within an allowed set. Examples of an optimization methodare Newton's method and the Levenberg-Marquardt algorithm.

The usage of the image processing may include, but is not limited to,detecting or recognizing the real environment or a part of it, visualsearch, classification problems, estimating the pose of an imagecapturing device, and/or tracking the real environment or a part of it.

The usage of the visualization may include, but is not limited to,delivering user's information associated to the real environment or apart of it, and/or enhancing user's visual perception by superimposingvirtual information or digital content to the real environment in acorrect perspective.

The visual properties of the digital information or the real environmentare, but are not limited to, a spatial property, behavior, color and/ortype. Behavior may include, but is not limited to, static (in the senseof remaining stationary), moving, and the way of moving.

The type of the digital information as described herein may include, butis not limited to, text, drawings, and videos.

The display devices as described herein are the devices displaying theoverlay of the digital information and the real environment to a user,such as monitors (e.g., LCD monitors), optical-see-through displays(e.g., optical-see-through head mounted displays), etc.

While the invention has been described with reference to exemplaryembodiments and applications scenarios, it will be understood by thoseskilled in the art that various changes may be made and equivalents maybe substituted for elements thereof without departing from the scope ofthe claims. Therefore, it is intended that the invention not be limitedto the particular embodiments disclosed, but that the invention willinclude all embodiments falling within the scope of the appended claimsand can be applied to various application in the industrial as well ascommercial field.

What is claimed is:
 1. A method of image processing in an augmentedreality application, comprising: receiving, by the augmented realityapplication, an image of a real environment, the application beingexecuted by one or more processors; performing image processing in theaugmented reality application using the image, the image processingemploying (i) a visualization of digital information overlaid onto oneor more of the image and the real environment, and (ii) a vision-basedprocessing or tracking technique, wherein the visualization is presentedusing a display device, and wherein a threshold of a re-projection errorof the digital information is adjusted according to a pixels-per-inch(PPI) parameter of the display device; and adjusting one or more of aparameter and an operating flow of the vision-based processing ortracking technique.
 2. The method according to claim 1, wherein thevision-based processing or tracking technique comprises vision-basedtracking, detection, localization, or classification.
 3. The methodaccording to claim 1, wherein the parameter of the vision-basedprocessing or tracking technique includes one or more of: a variablehaving a value, a number of features in the image matched with a set ofreference features, and a number of features in the image to be used fora pose estimation of a camera capturing the image.
 4. The methodaccording to claim 1, wherein the operating flow of the vision-basedprocessing or tracking technique includes a plurality of components,wherein at least one combination of the components is employed in thevision-based processing or tracking technique, and wherein the pluralityof components comprises one or more of: one or more sensors measuring aspatial property of an object in the image or the real environment, andone or more mathematical methods used in computer vision or imageprocessing.
 5. The method according to claim 4, wherein, when theplurality of components comprises the one or more sensors, the objectcomprises a space in one or more of the image and the real environmentand the spatial property is a property relating to the space.
 6. Themethod according to claim 5, wherein the property relating to the spacecomprises a geometry relating to the space, and the geometry describesone or more of: a shape relating to the space, a symmetry relating tothe space, a planarity relating to the space, a geometrical sizerelating to the space, and a density relating to the space.
 7. Themethod according to claim 4, wherein, when the plurality of componentscomprises the one or more mathematical methods, the one or moremathematical methods include one or more of: a linear equation solver, acost function, and an optimization method.
 8. The method according toclaim 1, wherein the display device comprises a device displaying one ormore of the digital information and the real environment to the usersuch that the user can visually perceive the real environment throughthe device.
 9. The method according to claim 1, further comprising:quantifying a performance of the vision-based processing or trackingtechnique based on a similarity measure of sum-of-squared differences,wherein the similarity measure is defined, at least in part, by asimilarity threshold value; and adjusting the similarity thresholdvalue.
 10. The method according to claim 1, wherein, when the digitalinformation includes a spherical object, the vision-based processing ortracking technique includes optimizing at least one parameter describinga translation of the spherical object, and not considering at least oneparameter describing a rotation of the spherical object.
 11. The methodaccording to claim 1, wherein the visualization includes an animation ofthe digital information, and wherein, when the animation of the digitalinformation includes a movement of the digital information, thevision-based processing or tracking technique does not include a stablepose estimation of a camera capturing the image.
 12. The methodaccording to claim 11, wherein a number of feature correspondencesemployed to estimate the camera pose is decreased.
 13. The methodaccording to claim 1, wherein adjusting one or more of a parameter andan operating flow of the vision-based processing or tracking techniqueis based on one or more of: a usage of the image processing; a usage ofthe visualization; a visually perceivable property of one or more of thedigital information and the real environment; a property of the displaydevice employed in the visualization; and a manner in which a user isviewing the visualization.
 14. The method according to claim 13,wherein, when the adjustment of one or more of the parameter and theoperating flow is based, at least in part, on the usage of the imageprocessing, the usage of the image processing includes one or more of:detecting at least one part of the real environment, performing a visualsearch of at least one part of the image or the real environment,detecting classification problems associated with the image or the realenvironment, estimating a pose of a capturing device capturing theimage, or tracking at least one part of the real environment.
 15. Themethod according to claim 13, wherein, when the adjustment of one ormore of the parameter and the operating flow is based, at least in part,on the usage of the visualization, the usage of the visualizationincludes one or more of: delivering information to the user that isassociated with at least one part of the real environment; and enhancingthe user's visual perception of one or more of the digital informationand the real environment by superimposing the digital information ontothe real environment in a correct perspective.
 16. The method accordingto claim 13, wherein, when the adjustment of one or more of theparameter and the operating flow is based, at least in part, on thevisually perceivable property, the visually perceivable propertyincludes one or more of: a spatial property associated with one or moreof the digital information and the real environment, a symmetryassociated with one or more of the digital information and the realenvironment, a behavior associated with one or more of the digitalinformation and the real environment, a color associated with one ormore of the digital information and the real environment, a type of thedigital information, and a type of the real environment.
 17. The methodaccording to claim 16, wherein, when the visually perceivable propertyincludes the behavior, the behavior comprises data describing a positionor a change in the position of at least one part of the digitalinformation or the real environment, and wherein, when the visuallyperceivable property includes the type of digital information, the typeof digital information includes one or more of a text, a drawing, and avideo.
 18. The method according to claim 13, wherein the manner in whichthe user is viewing the visualization comprises one or more of eyegazing and eye focusing.
 19. A non-transitory computer readable mediumcomprising instructions for image processing in an augmented realityapplication, which when executed by one or more processors, causes theone or more processors to: receive an image of a real environment;perform image processing in the augmented reality application using theimage, the application being executed by the one or more processors, andthe image processing employing (i) a visualization of digitalinformation overlaid onto one or more of the image and the realenvironment, and (ii) a vision-based processing or tracking technique,wherein the visualization is presented using a display device, andwherein a threshold of a re-projection error of the digital informationis adjusted according to a pixels-per-inch (PPI) parameter of thedisplay device; and adjust one or more of a parameter and an operatingflow of the vision-based processing or tracking technique.
 20. A systemfor image processing in an augmented reality application, comprising:memory storing data, the data including one or more computer-readableinstructions; a display device coupled to the memory; and one or moreprocessors coupled to the display device and the memory, the one or moreprocessors being configured to execute the computer-readableinstructions to: receive an image of a real environment; perform imageprocessing in the augmented reality application using the image, theapplication being executed by the one or more processors, and the imageprocessing employing (i) a visualization of digital information overlaidonto one or more of the image and the real environment, and (ii) avision-based processing or tracking technique, wherein the visualizationis presented using the display device, and wherein a threshold of are-projection error of the digital information is adjusted according toa pixels-per-inch (PPI) parameter of the display device; and adjust oneor more of a parameter and an operating flow of the vision-basedprocessing or tracking technique, wherein the adjustment of one or moreof the parameter and the operating flow is based on one or more of: ausage of the image processing; a usage of the visualization; a visuallyperceivable property of one or more of the digital information and thereal environment; a property of the display device employed in thevisualization; and a manner in which a user is viewing thevisualization.